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ABSTRACT 



This paper describes a statewide sustained effort to improve 
student learning by adopting and adaptinj promising research-based 
educational practices. The concept of an Instructional Management 
System (IMS) with desired learner outcomes determining both 
instruction and testing was sugges\.ed to local school districts 
begi-ining in 1980. In 1985, legislation provided both funds and 
authority to engage large groups of teachers to create criterion- 
referenced tests in reading/ language arts, mathematics, science 
and social studies /civics for various age groups. The refinement 
process which resulted in the Missouri Mastery and Achievement 
Test batteries is described. The achievement gains of large 
populations of students as measured during the first tv»»o years for 
which comparative data were available are also reported. The 
process of periodic review and revision is also briefly described. 
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Implementing and Assessing a Large Scale, School 

Improvement Project 



Richard L. King 
Missouri Department of Elementary and 
Secondary Education 

The school improvement project I shall describe for you today 
is occurring in Missouri. Missouri is a state in the center of 
the United States. The population of Missouri is 5,103,000. The 
kindergarten through grade 12 public school enrollment in Missouri 
is about 890,000 students. 

Missouri has two urban centers, St. Louis and Kansas City. 
There are several middle-sized cities and many small villages. 
The culture of the people ranges from the Ozarks "hillbilly" to 
the high tech corporate executive. There are 545 local school 
districts and about 3000 school buildings v.'here about 50,000 
teachers work. 

In the late 1970 's it became embarrassingly evident that our 
schools were not meeting the needs of our society. This was true 
throughout the United States. American schools were structured 
and taught so that the students were classified into levels of 
learning early in their school careers. Through this selection 
process, a few students received excellent educations and others 
demonstrated only moderate to poor learning. Until the 
mid-twentieth century this type of educational structure served 
the economic needs of our society quite well. We needed 15 to 20 
percent of our people well educated to serve as leaders in all 
aspects of our society. We needed many to do menial and 
physically demanding work which required much less high level 
skill or knowledge. 

After World War II/ it became increasingly apparent that a 
much higher percent of our population needed more and better 
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knowledge and competence to function productively in the more 
technically demanding society. Also, the advances in travel and 
communication- -the shrinking world--introduced social problems 
that required more intercultural understandings and social skills. 
This background information is not new to those of the western 
world; we have all experienced it to some degree. 

The Missouri Department of Elementary and Secondary Education 
(DESE) has decided over the last ten years to attempt to reverse 
the existing sad state of affairs. In an effort to select a long 
term education improvfiinent effort, the Department studied the 
available research to determine what interventions or practices 
contribute most to increasing the "effect size" of achievement 
gains. Drawing on the research data pertaining to curriculum 
alignment, instructional alignment (Cohen, 1987, 1988), mastery 
learning (Guskey, 1985; Block, 1988), and cooperative learning 
(Johnson and Johnson, 1985, 1987) we have formulated a 
research-based, outcome- based educational system which we call 
Instructional Management System (IMS). (Mo. DESE, 1986; 

The IMS concept is based on a large body of educaf *al 
research, which can be summarized in the following two premises: 

1. Ml students can learn what we teach to a satisfactory 
level — if teachers believe they can and if the school is 
organized to provide adequate time for each student to 
learn. (Carroll, 1963; Bloom, 1968) 

2. For students to be self -motivated to continue learning 
throughout their school careers and throughout life, they 
must spend more than half their time working on tasks at 
which they experience high degrees of success. This 
implies that each student should encounter learning tasks 
at a rate no faster than he/she can master. Learning 
deficits should not be allowed to accumulate. (Ames & 
Ames, 1984; Guskey, 1985) 

In June, 1981, the Missouri State Board of Education adopted 
as a top priority the implementation of an IMS in every district 
of the state. The remarkable feature o-f this action was that it 
made IMS a primary Board priority for tne "decade of the 80' s." 



Seldom do policvmaking boards understand the time required to 
bring about substantial change — especially when the constituency 
is largo and the change is not mandatory. (Mo. D^SE, 1986) 

During the summer of 1981/ Department staff conducted six 
day-long conferences on IMS in Jefferson City, the state capitol. 
A total of about 800 school personnel, mostly administrators, 
attended these meetings. They were introduced to the IMS concept 
and informed of the renewed state commitment to outcome-based 
schooling. Their response ranged from enthusiasm to skepticism, 
both toward the idea of IMS and toward the long-term state 
commitment to significant change. One superintendent, who was 
a^^ked by a colleague how he planned to respond to the call for an 
IMS, said, "I'm just going to hunker down until it goes away." 

During this same time, two or three of the Department staff 
members conducted meetings at all the state colleges and univer- 
sities. The purpose of these meetings was to explain to teacher 
educators the IMS concepts and to elicit their support by incor- 
porating these concepts into teacher and/or administrator training 
programs. We also held a day-long meeting in Jefferson City for 
teacher educators to inform them of our proposed emphasis on IMS. 
This meeting was attended by 87 college and university professors. 

These efforts have had mixed results. Relatively few teacher 
educators have provided thorough training in outcome-based or 
mastery learning practice; still fewer have incorporated these 
concepts into their own classes. Teachers, for the mc part, 
find the development of an IMS or organizing for maste* learning 
quite difficult. The demand for technical assistance is high. 
Even teachers who have recently completed their training (whether 
in Missouri institutions or in institutions of other states) feel 
inadequate when it comes to instructional alignment. They are 
unaware of the subtleties of directing instruction toward the 
outcome assessment. This has not been demanded of them nor taught 
in teacher training. Teachers' experience and training has often 
resulted in expectations that instruction causes a normal 
distribution in student achievement when the exact opposite of 
this is a major premise of an IMS. 
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To address this need for inservice traininC/ Department staff 
have conducted three or four week- long IMS workshops each suiraner 
at state universities for sjraduate credit. Enrollment at each 
workshop has been limited to 100. Applicants exceeding this 
number ha/e been turned away at nearly every scheduled workshop. 
Most of the workshop participants have come as local school 
district teams. Enrollment fees, paid by local districts or by 
individuals, have paid the costs of these workshops. In addition, 
during the school years since 1981, many local district staff 
development workshops have been conducted by the Department staff. 

The Board and Department staff have also tried to build 
public awareness and understanding, of the IMS concept. In recent 
years, for example, the Department has conducted an annual series 
of 10 "Regional Educational Conferences" in order to provide a 
forum for public discussion of a current educational issue. Each 
fall these meetings have attracted about 5,000 Missourians, 
including citizens, school board members, community leaders, 
parents, students, legislators and school district personnel. 

The meetings include large-group presentations by state 
education officials, small-group discussion sessions for 
participants, and an opportunity for participants to express their 
views on current topics. During 1981 and 1982, the Department's 
Regional Conferences focused on instructional management and 
outcome -based education. The same topic was featured two 
consecutive years to emphasize the importance of these issues aid 
to demonstrate a state-level commitment to the goal of promoting 
outcome-based .education in all public schools. 

To accelerate the change from a textbook driven curriculmn to 
an outcome driven instructional program, the Department has 
established six networks of school districts committed to 
outcome "based education. 

All school districts in the state were invited to become 
network meiint)ers. Each participating district was asked tos 

(1) involve the entire district or at least one building at 
each level (elementary, junior /middle, or senior high) in 
implementing an outcome based education program? 
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(2) agree to support financially teacher attendance at a 
one-week siammer workshop and four all-day Saturday 
meetings for the upcoming school year and for the 1990-91 
school year; and 
(3^ confirm its commitment by completing a proposal form 
signed by both the superintendent and the president of 
the local board. 
Forty-one districts responded adequately and were accepted as 
network members. Each was assigned to one of six networks 
organized geographically around the State. 

The Department's commitment is to provide outstanding 
educational experts and leaders in the field of outcome based 
education to conduct the week-long workshops and to meet with 
teachers on the scheduled Saturdays during the two school years. 
The Department's curriculum consultants and other staff members 
are also committed to providi.ig ongoing follow-up and technical 
assistance to the schools. 

The first round of v;orkshops is currently in progress. The 
workshops are being conducted by Dr. Jim Block of the University 
of California - Santa Barbara; Dr. Alan Cohen and Dr. Joan Hyman, 
University of San Francisco; Dr. Tom Guskey, University of 
Kentucky; and Dr. Robert Burns of the Far West Regional 
Educational Laboratory. 

The more than 450 participating teachers are not only 
implementing outcome-bared education in their respective 
districts y they are als.. 'oeing trained to be expert trainers for 
other teachers- 

Further impetus toward an outcome -based approach to learning 
was provided by the 83rd Missouri General Assembly. In 1985 the 
legislature passed House Bill 463^ the "Excellence in Education 
Act of 1985." This Act dealt with all aspects of education 
simultaneously €kQm teacher training and certification to 
curriculiun standards and testing for students. Following is the 
complete text of Section 4 of the Act, which has provided a 
framework whereby efforts to j.mplement an IMS could be accelerated 
across the state: 
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H. B. 463 

SECTION 4 
(Emphases added.) 

I. The commissioner of education shall direct the department 
of elementary and secondary education to insure that all 
school districts have a program of pupil testing which 
shall test competency in the subject areas of English, 
reading, language arts, science, mathematics, social 
studies and civics. 

2. The department of elementary and secondary education 
shall ident if y kev skills within the subject areas 
contained in subsection 1 of this section which shall 
provide the foundation for the local school district's 
testing program. The department of elementary and 
secondary education may not set maximal testing 
standards. 

3. Each local school district shall have a testing program. 
District testing programs may include minimum promotion 
standards and shall give due consideration to the 
resea. ch on the influence of cultural diversity on 
testing performance. 

4. The testing program of each local school district shall 
include, but shall not be limited to, criterion- 
rf^ff^renced tests approved by the depar^mf^nt of elementary 
and secondary education . This testing program shall test 
all students at periodic grade levels. The testing 
program may test students and shall identify areas for 
instructional improvement. The department of elementary 
and secondary education may develop criterion-referenced 
tests and assist districts with their testing programs 
upon the district's request. 

5. The department of elementary and secondary education 
shall develop or select tests which measure student 
performance on minimum key skills, and shall annually 
administer such tests to a randomly selected, statewide 
sample of public school students. 

6. Each local school district shall provide testing 
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information upon request to the department of elementary 
and secondary education. 
7 . The department of elementary and secondary education 
shall annually report to the general assembly composite 
pupil testing information. 

Developing Core Competencies /Key Skills 

As soon as it became apparent that the Excellence in Educa- 
tion Act would pass and be signed into law, the Commissioner of 
Education appointed an Oversight Committee. This committee 
consisted of more than 100 persons representing all professional 
education organizations, labor leaders, agriculture and business 
leaders and legislators. After sharing DESE proposed plans for 
implementing Section 4 of H.B. 463 and receiving several sugges- 
tions from members of the Oversj.ght Committee, statewide commit- 
tees of teachers, subject area specialists and school administra- 
tors were appointed to recommend the "core competencies and key 
skills" to meet the mandate of the law. 

Several critical decisions were made before the key skills 
were selected: 

1. The decision was made to start at grade 10. This is 
generally the end of the "common core" curriculum exper- 
ienced by all students. In addition, we decided to list 
as "key" those skills in each subject area which would 
undergird any elective courses a student might choose at 
grades 11 and 12. Finally, the tenth grade was selected 
because most of the high school dropouts in Missouri 
occur during grades 11 and 12. Thus, we could test a 
greater percentage of students than if we waited until 
grade 12 to test. 

2. We decided to "work down" from grade 10. What does a 
student need in grade 9 as prerequisite to doing well in 
grade 10? What in grade 8?, etc. 

3. Another decision was made to go only as low as grade 2 in 
the list of core competencies and key skills. .Irade 1 
skills were to be listed along with those for children 
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ages four through six. These are now being developed. 

4. It was also decided that we should include iroportant 
outcomes that cannot be measured by a paper/pencil test 
in the key skills list. These would, of necessity, be 
assessed locally. 

5. Because the legislation calls for testing of a statewide 
sample of students at "periodic" grade levels, we chose 
to prepare tests for grades 3, 6, 8 and 10 first. 

6. Student writing would be assessed each year on a sample 
of students at grades 6, 8 and 10. Holistic scoring 
would be done, for the state sample, by teachers who have 
been trained in this approach of assessing writing. 

7. These tests would not be "grade level" tests in the usual 
sense of the word. They would not only measure what had 
been taught the year the test was taken, but they would 
also sample those skills, competencies and knowledge that 
a student had accumulated as of early April each year. 
This was done to spread accountability for student 
learning beyond the individual teacher and to include the 
building administrator /s. How well is the principal 
managing the resources in his/her building to assure 
every student a quality school experience? 

More than 300 educators, from primary grade teachers through 
college professors, were involved in developing Missouri's key 
skills. Our ''Core Competencies and Key Skills" (formally 
published in the fall of 1986 and revised in 1989) do not include 
all that should be learned through grade 10. Instead, the state 
list is a skeleton for local districts to "flesh out" as they see 
fit. The skeleton analogy is rather appropriate since everyone's 
skeleton is very similar when viewed in an x-ray. It is only when 
the necessary flesh and blood are added that the individual 
personal characteristics make us individuals. This allows for 
much local curriculum autonomy. Also, a skeleton is connected 
from the "toe bone" to the "head bone;" similarly, our key skills 
are also vertically articulated. 

Several other concepts that research has demonstrated as 
adding to the "effect size" of achievement gains have been 
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incorporated into Missouri's plan for achievement gains: 

" 1. The key skills were developed to build sequentially the 
concept of "enhanced prerequisites." (Leyton, 1983) If 
the teachers of sequential courses agree on the prerequi- 
sites needed for the subsequent course in a specific 
school subject, then each teacher of the first course can 
insure that by the end of the term most of the students 
have mastered these prerequisites before going on to the 
subsequent course in the subject. 

2. The importance of home participation and support and that 
of enhanced cues and reinforcement through frequent 
feedback are stressed. Learner outcomes and mastery 
criteria are clearly spelled out before instruction 
begins. Both the student and his/her parents are 
provided this information. After initial instruction and 
the first formative test the student is provided feedback 
as to what he has learned well and what be still needs to 
work on. This coupled with the mastery learning process 
has been 'jhown to have a possible effect size of two 
Sigma. (Tenenbaurn, 1932) 

3. Teaching for higher order thinking has been shown to add 
to the total effect size, also. (Mevarich, 1980) 
Cognitive psychologists have classified more than half of 
the core competencies /key skills as being above the level 
of "comprehension" on Bloom's Taxonomy of Educational 
Objectives. (Bloom, 1956) 

The Missouri Department of Elementary and Secondary Education 
sponsored two major statewide conferences: "Beyond Effective 
Schools" in 1986 and "Beyond Effective Schools II" in 1987. These 
were planned to let school district teams learn more about mastery 
learning, cooperative learning and OBE. Major presenters at these 
conferences have been Carol Barber, James Block, John Champlin, 
Alan Cohen, Thomas Guskey, David Johnson and Susan Loucks-Horsley . 
Care was taken to have each of these presenters come to the 
Department for pre-conf erence consultation to insure that the 
focus of the conference was sharp and clear. Several of these 
speakers have returnee to the State after the meetings to provide 
workshops in their area of expertise. 
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Draft copies of the proposed Core Competencies and Key Skills 
were distributed and discussed at fourteen regional curriculum 
conferences across the state which were attended by more than 4000 
teachers and administrators during the fall, 1985. Suggestions 
obtained at these meetings were incorporated in the final draft of 
the skills. 

As the Core Competencies/Key Skills were being refined, the 
DESE met with representatives from every major test publisher in 
the United States to see if their firms could get tests developed 
which would meet Missouri's needs. There were two important 
reasons for DESE to reject all proposals by commercial testing 
companies to produce the tests: first, the items in their item 
banks which they showed DESE were almost all at the level of 
simple recall or memory and, second, the testing company 
representatives were so indoctrinated with the psychometric model 
of testing (normal curve as a design target) that they all 
proposed embedding a norm referenced set of items in the tests. 
The first design shortcoming violated the concept of curriculum 
and instructional alignment and the second was not consistent with 
the goal of mastery for all students. The DESE was committed to 
developing talent as opposed to selecting talent. 

Developing the Tests 

When the Department decided to develop its own tests, which 
were to be called the Missouri Mastery and Achievement Tests 
(MMAT), a contract was negotiated with the Center for Educational 
Assessment (CEA) at the University of Missouri, Columbia, to 
assist in organizing, developing, refining and finally, providing 
test scoring and reporting strategies. 



Insert Figure 1 about here 



A test development effort such as this requires large numbers 
of individuals to complete specified tasks at each stage of test 
development. While many individuals were involved in developing 
the MMAT, the groups described below were formally identified to 
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perform the functions indicated. 

Widespread involvement of Missouri educators ensured that the 
tests were crafted \7ith a firsthand knowledge of the state's 
school children and a special sensitivity to the educational 
growth of its young people. After the test objectives were 
determined, creation of the test battery itself required contribu- 
tions by hundreds of Missouri educatcrs at every level and 
scrutiny and revision of numerous drafts by CEA and the Missouri 
Department of Elementary and Secondary Education. 

More tt.an 200 teachers and administ itors from schools 
throughout Missouri were involved in item writing. Another group 
of more than 200 teachers reviewed the items written by their 
colleagues for agreement with the Key Skills and for quality of 
the item stem and its choices. Teachers and administrators were 
involved in every phase of test development. In addition, a 
sample of students who had participated in the field trials was 
interviewed with structured-interview protocols to assess which 
features of the items had prompted their responses. 

Consultants from the Department of Elementary and Secondary 
Education were involved with test development at all stages. 
Department consultants worked with CEA staff, teachers, and 
administrators in defining test content and in item writing. 

The CEA staff supplemented its expertise with numerous 
consultants who worked on specific aspects of test development. 
A panel of five testing experts from across the country served as 
the Technical Advisory Committee and provided input on issues 
concerning test construction, scaling, equating, norming, item 
analysis, reliability, and validity. Members of that panel have 
participated individually in completing specialized tasks and 
studies during test development. A nationally recognized 
specialist in sampling design devised the sampling plans employed 
in this effort. A number of individuals examined items for bias. 
In short, CEA utilized the skills of no fewer than 20 nationally 
recogr.ized individuals to amplify its professional efforts. 
Finally, since state officials and leaders in business, industry, 
and education were involved as an Oversight Committee during the 
implementation of the Excellence in Education Act of 1985, the 
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implcmeritation of the Act was addressed by utilizing the services 
of the broadest sampling of the community involved in education- 

A committee composed of 12 professors, three each from 
reading, social studies, mathematics, and science, wiute content 
specifications for test items for each of the 688 skills in both 
phases of development. Particular skills were assigned according 
to an individual's content or ciirricular expertise. The 
professors were instructed to elaborate, interpret, and clarify 
each key skill so that its particular characteristics might be 
clearly and uniformly understood by everyone involved. These 
elaborations formed the initial draft of the test content 
specifications . 

Practitioners Develop the Test Items 

Next, approximately 220 teachers from all grades throughout 
Missouri were assembled to review and further refine the 
specifications drafted by the college content area specialists. 
Workshop participants were charged with reviewing specifications 
in terms of their applica'jility to classroom instruction. 
Teachers met in groups of five to seven for each grade level and 
for each content area. Curriculum consultants from the Department 
of Elementary and Secondary Education assisted the teachers in 
reviewinr specifications. The reviewers' comments and the 
exemplary items they produced were compiled to describe each skill 
according to content domain and grade levex. 

This version of the test content specifications was then 
reviewed by the college content area specialists. CEA editorial 
staff and Department consultants additionally reviewed and revised 
the specifications. Finally, each test content specification was 
reviewed by Department consultants and two to five teachers for 
specific content at a particular grade level. In total, more than 
1,000 reviews of test content specifications had occurred by the 
time they reached their final form. The final test content 
specifications were then printed and distributed to all school 
districts in Missouri. Many teachers use the specifications in 
preparing lesson plans. 
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A test item writing workshop was then conducted. As with the 
Test Specifications Conference, teachers were grouped according to 
their subject areas and grade levels. Test specifications 
for the grade level and the content area were available to each 
group. 

Two test item construction experts taught participants 
techniques of item writing during the workshops. In addition, 
consultants for the Department and from CEA worked closely with 
the teachers. 

Hundreds of test items obtained from public domain item banks 
were available to item writers during the workshop. The 
teacher/authors decided that these items were not appropriate for 
assessing the Key Skills, and that the content of the items 
themselves was not suitable. The items were abandoned in favor of 
developing original items referenced directly to the test content 
specif icc :ions and Key Skills. 

By the close of the workshops, 12 to 16 items for each of the 
688 test content specifications had been developed. This 
translated to approximately 9,600 items. 

Another group of approximately 240 teachers reviewed the 
items. Content specialists, including college professors, 
teachers, private consultants, curriculum consultants from the 
Department, and the CEA staff, reviewed the items. This process 
provided at least five reviews per item. Each systematic review 
required the use of a scorecard which included specific criteria 
relating/ to congruence between item and the key skill, 
appropriateness of the reading level for the grade level, wording, 
format, and appropriateness of distractors. These reviews were 
analyzed and the results were synthesized by CEA editorial staff 
and curriculum consultants from the Department of Elementary and 
secondary Education. Modifications of particular items were made 
when appropriate. In some instances, these exhaustive reviews 
resulted in the removal of test items from further consideration. 

Studies of the readability of items were completed by a 
reading specialist using standard formulas such as Flesch, 
Dale-Chall, SMOG, and FOG. For grades 2 and 3, a panel of 15 
classroom teachers reviewed the test items and the directions to 
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ensure that the syntax and vocabulary would be understood by 
children of this age. 

Item bias reviews, both judgmental and empirical, were 
undertaken. A panel with nationally accepted expertise in test 
item bias judged items for bias. Approximately 150 classroom 
teachers were asked to rate items systematically with specially 
constructed rating sheets to point out stereotyping or other 
objectionable wording which might bias items. Each item was rated 
approximately five times by classroom teachers who examined how 
each item might affect the way a child with a culturally or 
linguistically different background might respond. For each bias 
review, clearly delineated criteria were used. Items identified 
as biased were either discarded or flagged as needing editorial 
repair. Additionally, CEA editorial staff adjusted items to 
ensure a mix of genders in the name references in items, a mix of 
traditional and non-traditional gender roles, and a variety of 
ethnic backgrounds. 

The initial test item pool contained approximately 9,600 
items. The text alone for each phase required some 70,000 lines. 
More than 800 galleys were set. Five graphic artists completed 
approximately 3,000 graphics for the items. Just under one-third 
of the items required graphics, and a large number required text 
to accompany items. 

This exhaustive process of item development, review, and 
revision has resulted in a pool of items of exceptional quality. 
The language of each item had been clarified; its congruence to a 
specific skill had been verified, and its ability to be understood 
by students had been examined. Several of the guiding principles 
which insured the quality of the resulting test include: 

- Field participation. Teachers and subject area consultants 
established the content validity of the test. 

- Expert judges. Curriculum and measurement authorities 
reacted to each stage of the test development and field 
trial techniques. 

- A common- items equating design was used to ensure that the 
final forms for each level were indeed parallel. 

The final version of the MMAT consists of four forms of 
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thirty-four criterion-referenced tests which assess student 
performance on Key Skills in grades 2 through 10. The grade 2 
tests cover Language Arts/Reading and Mathematics. Subject tests 
for grades 3 through 10 include Reading/English/Language Arts, 
Mathematics, Science, and Social Studies/Civics. The battery 
consists of multiple-choice items with four options each. There 
are two parts to each of the subject tests; this allows students 
to be given a rest break during each subject test. 

Using the MMAT to Test School Improvement 

In the Spring of 1987, approximately 240,000 students in 
grades 3, 6, 8, and 10 were administered Form A. .In the Spring of 
1988, approximately 240,000 students in grades 3, 6, 8, and 10 
were administered Form B and approximately 200,000 students in 
grades 2, 4, 5, 7, and 9 were administered Form D. During the 
next two years more than 83 percent of Missouri's students are 
scheduled to take the MMAT in grades 2 through 10; the remainder 
are giving the test to the required minimum of four levels. 

The tests are administered in early April, scored, and the 
reports are returned to schools during the last two weeks in May 
before school is out for the year. 

Local school district reports include the following: 

1. Individual student report. 

This report lists, by subject, the Key Skills 
mastered and those not mastered, the student's scale 
score (on a scale of 100-500) and the district average 
scale score for that grade and subjeci- 

The scale score of 300 was arbitrarily set to 
reflect the state average performance for 1987. A sample 
of schools representing rural/urban, large/ small, high 
SES/low SES was randomly chosen (more than 5,000 students 
per grade level) and the same sampling process is used 
each year to obtain the State sample. 

2. Individual Student Score Labels are supplied to be 
affixed to the student's permanent record. 

The ITBS and the TAP are administered each year to 
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the State's Sample to provide comparable national percen- 
tile ranks. This also allows the State to provide 
Chapter I eligibility lists to each district. This label 
lists the student NCE. 
Pupil List Report. 

This report lists all students, by grade and 
subject, in a school building with mastery/ non-mastery of 
each Key Skill indicated. 
Grade Level Key Skill Report. 

This report presents the percentage of students 
mastering each Key Skill. There is a separate Grade 
Level Key Skill Report for each subject for a grade in a 
building and in a district. 
Grade Level Cluster Report. 

This report depicts in graphic form the performance 
of students in a specific building or a district. Each 
"x" corresponds to the average scaled score of the 
Students in a building or the district at that grade on a 
subject or cluster. The bar extending through the "x" 
represents the standard deviation and shows the degree of 
variation among the scores. 
Summary Report. 

This report provides information about an entire 
building's or district's performance at grades 2 through 
10 on each subject tested on the MMAT. The Summary 
Report, when compared to previous years' reports, can be 
a useful tool for monitoring changes over time in MMAT 
performances. 

District Summary Report by Quintiles. 

Quintiles for the State sample scores were estab- 
lished in 1987. This report displays what percent of 
students in a local district fall in each fifth of the 
state rankings established in 1987. This torm is 
especially valuable to monitor progress toward equity in 
learner outcomcjs — which is a goal of OBE. 
A school district may elect to receive other reports 
including summary data disaggregated by socio-economic 
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status, race, or sex, as well as output on computer 
diskette or tape. 
The results of the 1988 and 1989 administration of the MMAT 
demonstrated that the Department's optimistic expectations were 
not misplaced. All of th3 state's sample showed gains. Obviously 
all of the more than 450 school districts in Missouri aad not 
implemented all aspects of an IMS in all subjects and levels. 



Insert Table 1 about here 



The gains will more nearly approach what research has shown is 
possible as more districts adopt OBE practices. 

There are three components to any effective accountability 
system: 

1. There must be a clear set of desired outcomes; 

2. There must be a feedback system so that one can know how 
well these outcomes are being achieved; 

3 . There must be consecruences that matter . 

It is obvious that we now had the first two requisites of an 
accountability system. To provide for consequences that matter, 
we invited school districts to volunteer to pilot the use of 
achievement data as a factor in school district classification. 
Prior to this time school classification in Missouri had been 
based upon inputs only. We sought at least ten percent of the 
districts to volunteer. This would have been 45 districts. 
Instead we had 109 districts with total enrollments of 185,000 
which volunteered. Here is the reports of these 109 districts. 
There were about 15,000 students in each grade level reported. 
These districts reported gains of more than one sigma during this 
two year period in some cases. 



Insert Table 2 about here 



An even more crucial measure of how this entire process is 
bringing about improvement in the equity of student progress can 
be seen in the quintile reports. The State sample had exactly 20 
percent of the students in each quintile in 1987 because that is 
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how the quintiles were established. Two years later we can see 
that a higher percent of students are in the higher one- fifth and 
fewer in the lower. 



Insert Table 3 about here 



The 109 "Academic Achievement Districts" made even more 
difference. 



Insert Table 4 about here 



There are two interesting observations about this table: 
These districts had nearly state average performance in 1987 and, 
second, they have quite high percents of students in the top two 
fifths and remarkably lowered percents in the bottom fifth. Th5s 
is after only two years. 

When one looks at the state sample (more than 20,000 students 
each year) and see statistically significant gains at all four 
levels and in each four subjects, the question arises: What 
caused these gains? Mastery learning and cooperative learning 
were only common practice in fewer than half the districts of the 
state. The only changed practice which was common to all schools 
was curriculum and instructional alignment. The mean scores in 
1988 had moved far enough from the mean established in 1987 to 
cause a notable increase in the standard deviation from one year 
to the next in all cases. (Table 1) 

During school year 1988-89 we began the process of reviewing 
and revising our Core Competencies and Key Skills. Every school 
in the state was given an opportunity to suggest changes and large 
committees of teachers were drafted to consider the suggestions 
and help revise the learner outcomes. 

The same test development process that we used for the 
original test has been underway during the current school year. 
This time we are developing new forms for grades 7-10. Next year 
we will develop new forms for grades 2-6. The 7-10 test items 
will be field tested during the Spring and Fall of 1990 and will 
not be used officially until Spring 1991. The elementary grades 
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test forms will be introduced the next year. 

Using the WIKJ! to Cause School Improvement 

^s students and teachers become more adept in outcome based 
instructional practice, and as students experience more mastery 
learning and cooperative learning we should expect to see 
continued substantial gains in school achievement. The Core 
competencies and Key Skills were reviewed and revised during 1989. 
Three new forms of the MI^T have been developed by teachers as the 
first forms were. These new test forms will be piloted and field 
tested during 1990 and used for the 1991 testing program. The new 
test forms are to be equated to the old forms to prevent any 
marked discontinuity in trend data. 

several lessons have been learned by Missouri's experiences 
in the quest for improved student learning: 

1. Research-based practices which show Pd^icationally signif- 
icant promise, such as outcome based educational decision 
making, mastery learning and instructional alignment, are 
the only practices that justify a major state effort. 

2. The role "^f the State Board of Education and the 
commissioner of Education is extremely important to 
successful statewide changes. Their long term public 
commitment to instructional management has been crucial 
to the gains made in Missouri schools. 

3 . The credib lity and practitioner ownership of a process 
is positively related to the breadth of involvement of 
those most affected by the process. 

4. Many change efforts fail because the novelty of the 
change wears off and because new ideas intrude before 
sufficient time is devoted to the original practice to 
see it successful. Focus over time is needed in staff 
development to equip school staffs to incorporate changes 
effectively. 
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SUBJECTS AND CMJSTF.RS 



Reading or English/ 
Language Arts 


Mathematics 


Science 


Social Studies/ 
Civics 


Language Arts 


Understanding 
Numbers 


Life Science 


Geography 


Reading 


Computation 


Earth Science 


History 


Writing 


Measurement 
and Geometry 


Physical 
Science 


Government 




Interpretation 

and 
Application 


Cross- 
Disciplinary 


Economics 
Civics 



>Mgure 1 - Clusters of key skills included in the Missouri Core Competencies/Key Skills by 
subject area 
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r.raHft r Forms A. B and C Smtistics Qradp 8, Fonns A. B and C StaMs^ips 

Snrinp 1987. 198S & 1989 Administrations Spring 1987. 1988 ^ 19^V Administmlipn;; 



Number 
Subject of Items 


Scaled Score 


Scaled Score 


Standard IRT Number 
Means Deviation Reliability Subject of Items 


Standard IRT 
MpiinQ Deviation Reliability 


Reading 84 
Mathematics 68 
Science 64 
Social Studies 56 


Forms: Forms: Forms: 
ABCABCA BC 


Forms: Forms: Forms: 
A BCABCA B C 


300 315 328 
300 318 323 
300 311 344 
300 325 337 


65 68 73 
65 77 74 
65 72 79 
65 73 71 


.948 .941 .933 Reading 108 
.929 .916 .910 Mathematics 100 
.909 .895 .888 Science 72 
.924 .905 .904 Social Studies 72 


300 322 326 65 70 71 .951 .950 .946 
300 336 33:> 65 84 84 .949 .953 .954 
300 337 345 65 88 82 .901 .906 .898 
300 325 325 65 77 75 .941 .937 .934 



4789 5706 5378 " =5346 4733 5258 



firade 6. Form s A. B and C Statistics 
Spring 1987. 198 8 & 1989 Administrations 



Number 
Subject of Items 


Scaled Score 


Standard IRT 
Means Deviation Reliability 


Reading 92 
Mathematics 104 
Science 92 
Social Studies 84 


Forms: Forms: Forms: 
ABCABCA BC 


300 308 321 
300 332 339 
300 332 344 
300 321 322 


65 72 70 
65 77 83 
65 84 83 
65 77 78 


.937 .937 .929 
.955 .956 .959 
.914 .926 .926 
.943 .942 .943 



n = 5138 5152 4903 



Ciradc 10. Forms A. B and C StaUstics 
Spring 1987. 1988 A 1989 Administrations 



Number 
Subject of Items 


Scaled Score 


Standard IRT 
Mcims Deviation Reliability 


Reading 112 
Mathematics 92 
Science 80 
Social Studies 100 


Forms: Forms: Forms: 
ABCABCA BC 


300 315 324 
300 313 330 
300 326 322 
300 314 311 


65 72 76 
65 72 82 
65 82 73 
65 70 72 


.948 .940 .946 
.951 .952 .956 
.891 .892 .887 
.957 .952 .943 



n:=5645 5023 4723 



Note - All gains are significant at the .0001 level from Form A to B; many are significant at the .0001 level from Form B to C. 

Table 1 - The standard error range is from .9 to 1 .2 2 g 
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Academic Achievement Districts: 1987, 1988 and 1989 



^Subject 
Grade >^ 


READING 
87 88 89 


MATH 
87 88 89 


S 

87 


CIENCI 
88 


!: 

89 


SO 
87 


CIAL SI 
88 


r. 

89 


3 


306 


334 


365 


304 


349** 


365 


307 


335 


418 


314 


340 


393 


6 


298 


323 


350 


303 


379* 


404 


305 


393* 


416 


304 


358** 


360 


8 


310 


349** 


353 


317 


387** 


390 


314 


385** 


412 


314 


362** 


358 


10 


304 


327 


355 


309 


328 


372 


308 


356** 


356 


305 


329 


328 



r ♦ one 



First year 
gain data 



Sigma gain, sigma for the baseline (300) is approximately 65. 



^ *♦ gains from .56 to .83 sigma 



Table 2 . Scale score aalns recorded for 109 Missouri school districts which volunteered to use achievement 
Table 2 ^ • ^^^Jl^^^jj^^ j;)^ ^.^^^ard. These districts have total enrollments of 185,000 students. 

There were approximately 15, 000 totaled at each grade level. 
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MISSOURI MASTERY AND ACHIEVEMENT TESTS 



ACADEMIC ACHIEVEMENT REPORT 



State Sample 1988 and 1989 









GRADE 3 












GRADE 6 








READ 


MATH 


SCI SOC.ST 


READ 


MATH 


SCI SOC, ST 




88 


89 


88 


89 


88 


89 


88 


89 




88 


89 


88 


89 


88 


89 


88 


89 


HIGH 


28 


35 


30 


32 


26 


44 


32 


40 


HIGH 


24 


30 


36 


39 


37 


43 


32 


33 






21 


21 


20 


22 


23 


22 


24 


25 




19 


21 


21 


20 


19 


20 


20 


20 






18 


17 


18 


18 


16 


13 


18 


16 




19 


19 


17 


16 


15 


16 


16 


17 






17 


14 


15 


14 


17 


11 


13 


11 




19 


17 


14 


13 


14 


12 


16 


15 




LUW 


15 


12 


17 


14 


18 


10 


13 


09 


LOW 


18 


13 


12 


11 


15 


10 


15 


15 




Number 


1988 • 5706 












Number 


19«8 - 5152 












Tested 


1989 - 5378 












Tested 


1989 - 4903 


















GRADE 8 












grade:io 








READ 


MATH SCI 


SOC. ST 


READ MATH 


SCI SOC.ST 




88 


89 


88 


89 


88 


89 


88 


89 




88 


89 


88 


89 


88 


89 


88 


89 


HIGH 


31 


33 


40 


39 


38 


43 


34 


34 


HIGH 


27 


32 


27 


34 


34 


32 


26 


25 






22 


22 


18 


19 


20 


21 


20 


21 




21 


20 


21 


22 


20 


21 


22 


20 






18 


1« 


16 


15 


15 


15 


17 


Id 




18 


18 


18 


17 


16 


17 


20 


20 






16 


16 


14 


13 


13 


12 


14 


13 




17 


15 


17 


15 


13 


15 


17 


17 




LOW 


13 


12 


13 


14 


14 


09 


15 


14 


LOW 


16 


15 


17 


13 


16 


15 


16 


18 




Number 1988 


•4733 












Number 


1988-5023 












Tested 1989 


-5258 












Tested 


1989 . 4723 













Table 3 - 1988 and 1989 percent of students scoring in each fifth established using 1987 data. 



ACADEMIC ACHIEVEMENT DISTRICTS: 1987, 1988 & 1989, MMAT 



ERIC 



READ 



HIGH 



LOW 



IIICH 



LOW 



o7 


OO 


QQ 
07 


23 


40 


55 


21 


22 


20 


18 


16 


12 


18 


13 


8 


29 


9 


6 


READ 


87 


88 


89 


25 


45 


47 


20 


23 


20 


20 


15 


11 


17 


11 


11 


18 


6 


7 



(iRADK3 



MATH 



SCI 



87 


88 


89 


25 


45 


54 


19 


22 


19 


19 


13 


13 


17 


10 


8 


20 


10 


6 



87 


88 


89 


24 


38 


77 


21 


22 


12 


19 


16 


5 


17 


13 


3 


19 


11 


2 



GRADE 8 



MATH 



SCI 



87 


88 


89 


28 


60 


60 


22 




16 


19 


11 


8 


16 


8 


8 


IS 


4 


7 



27 


59 


69 


19 


17 


15 


22 


11 


4 


16 


6 


4 


16 


7 


3 



roc. ST. 



87 


88 


89 


27 


38 


70 


21 


25 


15 


17 


19 


7 


16 


11 


5 


19 


7 


2 


SOC. ST. 


87 


88 


89 


28 


50 


50 


20 


20 


19 


19 


13 


12 


15 


9 


9 


18 


8 


10 



READ 



HIGH 



LOW 



HIGH 



LOW 



fi7 
or 


fUl 
OO 


07 


21 


32 


45 


20 


20 


21 


19 


18 


14 


19 


17 


12 


21 


13 


8 


READ 


87 


88 


89 


24 


34 


48 


21 


20 


17 


18 


18 


11 


19 


16 


11 


18 


12 


11 



GRADE 6 



MATH 



S(T 



87 


88 


89 


87 


88 


89 


22 


58 


65 




24 


64 


71 


19 


20 


16 




20 


16 


13 


19 


12 


9 




18 


9 


8 


18 


7 


6 




18 


6 


5 


22 


3 


4 




20 


5 


4 



GRADE 10 



MATH 



SCI 



87 


88 


89 


87 


88 


89 


25 


33 


53 




25 


50 


50 


21 


22 


19 




21 


18 


19 


19 


18 


9 




19 


12 


10 


15 


17 


9 




17 


9 


10 


18 


12 


8 




18 


11 


9 



Table 4 - Average percent of students in each fifth of the distributions of the 109 "academic achievement" districts. 
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SOC. ST. 



87 


88 


89 


23 


50 


50 


20 


20 


18 


18 


13 


12 


20 


10 


10 


19 


7 


10 


SOC. ST» 


87 


88 


89 


24 


32 


35 


21 


22 


19 


19 


19 


14 


18 


10 


14 


18 


12 


16 
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